Method and system for the storage and retrieval of web-based education materials

ABSTRACT

A system is provided that automatically digitally captures lecture presentation slides and speech and stores the data in a memory. This system also prepares this information for Internet publication and publishes it on the Internet for distribution to end-users. The system generally comprises three main functions: (1) capturing the lecture and storing it into a computer memory or database, (2) generating a transcript from the lecture and the presentation slides and automatically summarizing and outlining the transcripts, and (3) publishing the lecture slides image data, audio data, and transcripts on the Internet for use by client computers. The system synchronizes the slide image data, audio data and the transcripts, and the clients can view and search the published lecture presentation from a slide image projector to a digital camera for digital image data capture.

BACKGROUND

1. Field of the Invention

The present invention generally relates to a data processing system fordigitally recording lectures and presentations. More particularly, itrelates to the conversion of these lectures with little intervention toa standard Internet format for publication.

2. Related Art

The majority of corporate and educational institution training occurs inthe traditional lecture format in which a speaker addresses an audienceto disseminate information. Due to difficulties in scheduling andgeographic diversity of speakers and intended audiences, a variety oftechniques for recording the content of these lectures have beendeveloped. These techniques include videotapes, audio tapes,transcription to written formats and other means of converting lecturesto analog (non-computer based) formats.

More recently, with the advent and growing acceptance of the Internetand the World Wide Web, institutions have started to use thiscommunication medium to broadcast lectures. Conventionally, in order tocreate a Web-based lecture presentation that utilizes 35-mm slides orother projected media and that includes audio, a laborious process isnecessary. This process involves manually removing each slide anddigitizing it and manually recording and digitizing the audio into aWeb-based format. In addition, to complete the lecture materials, eachslide must be manually synchronized with the respective portion ofaudio. Thus, the entire process of converting a lecture into a formatthat can be published on the Internet is labor intensive, time-consumingand expensive.

One technological challenge has been allowing audio/visual media to bemade available on relatively low bandwidth connections (such as 14.4kilobits/second modems). Native audio and visual digital files are toolarge to receive in a timely manner over these low bandwidth modems.This technological challenge becomes prohibitive when one attempts totransmit a lecture over the Internet, which requires slide updates whilemaintaining simultaneous audio transmission. To this end, RealNetworks™, Microsoft™, VDOlive™ and several other companies havepioneered and commercialized a variety of techniques which allow forcontinuous, uninterrupted transmission of sound and images over theInternet, even over low bandwidth connections. This format, known as“streaming”, does not require the end-user to obtain the entire audio orvideo file before they can see or hear it. Recently, Microsoft hasprovided a standard media format for Web-based multimedia transmissionover the Internet. This standard is called the “Active Streaming Format”(ASF). The ASF Format is further described at the Internet websitehttp://www.microsoft.com/mind/0997/netshow/netshow.htm, which isincorporated herein by reference.

Furthermore, a variety of manufacturers (e.g., Kodak, Nikon, AGFA) havedeveloped technologies for scanning 35-mm slides and digitizing them.However, these systems have several disadvantages. Most significantly,they require removal of the slides from a slide carousel. Additionally,they require a separate, time-consuming scanning process (on the orderof several seconds per slide), and as a result, a lecturer cannot usethe scanners when giving a presentation due to the delay of scanningeach slide independently. Even further, they are not optimized forcapturing slide information for the resolution requirements of theInternet. These requirements are generally low compared with typicalslide scanners, since smaller file size images are desired for Internetpublishing. Finally, they are not designed to capture audio orpresentation commands (such as forward and reverse commands for slidechanges).

One device recently introduced to the market under the name “CoolPix300™” (available from Nikon of Melville, N.Y.) allows for digital videoimage and digital audio capture as well as annotation with a stylus.However, the device does not permit slide scanning and does not optimizethe images and audio for use on the Internet. Its audio recording isalso limited to a relatively short 17 minutes. Similarly, digitalaudio/video cameras (such as the Sony Digital Handycam series) allow forthe digital video and audio recording of lectures but have no directmeans of capturing slides. In addition, they are not set up to recordinformation in a manner that is optimized for the Internet. Generally,with these systems, the amount of audio captured is limited to about onehour before a new cassette is required to be inserted into the camera.

Although these conventional techniques offer the capability to transmiteducational materials, their successful deployment entails significantadditional manual efforts to digitize, synchronize, store, and convertto the appropriate digital format to enable use on the Internet. Addingto the cost and delay, additional technical staff may be required toaccomplish these goals. Furthermore, there is a time delay between thelecture and its availability on the Internet due to the requirement thatthe above processes take place. As such, the overall time required forprocessing a lecture using conventional methods and systems is five toten hours.

Another related technology for storing, searching and retrieving videoinformation is called the “Infomedia Digital Video Library” and isdeveloped by Carnegie Mellon University of Pittsburgh, Pa. However, thesystem under consideration will use previously recorded materials forinclusion into the database and thus makes no provisions for recordingnew materials and quickly transferring them into the database. Moreover,in this effort, there was no emphasis on slide-based media.

It is therefore desirable to provide a system that allows a presenter tostore the contents of a lecture so that it may be broadcast across theWeb. It is further desirable to provide a system that allows theefficient searching and retrieval of these Web-based educationalmaterials.

SUMMARY

Methods and systems consistent with the present invention satisfy thisand other desires by optimizing and automating the process of convertinglecture presentations into a Web-based format and allowing for theremote searching and retrieval of the information. Typically, systemsconsistent with the present invention combine the functionality of aprojection device, a video imaging element, an audio recorder, and acomputer. Generally, the computer implements a method for the conversionand enhancement of the captured lectures into a Web-based format that isfully searchable, and the lecture can be served immediately to theInternet.

A method is provided for recording and storing a lecture presentationusing slides and audio comprising the steps of initiating display of aslide image, capturing slide image data from the slide imageautomatically in response to the initiation and storing the slide imagedata in a memory. The method may further include the steps of recordingaudio signals associated with the slide image, capturing audio data fromthe audio signals, and storing the audio data in a memory.

The advantages accruing to the present invention are numerous. Forexample, a presenter of information can capture his or her informationand transform it into a Web-based presentation with minimal additionaleffort. This Web-based presentation can then be served to the Internetwith little additional intervention. The nearly simultaneous recording,storage and indexing of educational content using electronic meansreduces processing time from more than five hours to a matter ofminutes. Systems consistent with the present invention also provide ameans of remotely searching and retrieving the recorded educationalmaterials.

In one implementation, optical character recognition and voicerecognition software can be run on the slide data and audio recordingsto produce transcripts. Using additional software, these transcripts canbe automatically indexed and summarized for efficient searching.

The above desires, other desires, features, and advantages of thepresent invention will be readily appreciated by one of ordinary skillin the art from the following detailed description of the preferredimplementations when taken in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates hardware components of a system consistent withpresent invention;

FIG. 2 illustrates a mirror assembly used to redirect light from aprojection device to a digital camera consistent with the presentinvention;

FIG. 3 depicts the components of a computer consistent with the presentinvention;

FIG. 4 illustrates alternate connections to an overhead projector andLCD projector consistent with the present invention;

FIG. 5 shows input and output jacks on a system consistent with thepresent invention;

FIG. 6 is a flowchart illustrating a method for capturing a lectureconsistent with the present invention;

FIG. 7 is a flowchart illustrating a method for enhancing a capturedlecture consistent with the present invention;

FIG. 8 is a flowchart illustrating a method for publishing a capturedlecture on the Internet consistent with the present invention;

FIG. 9 shows an example of a front-end interface used to access thedatabase information consistent with the present invention;

FIG. 10 shows a schematic of a three-tier architecture consistent withthe present invention;

FIG. 11 shows an alternative implementation consistent with the presentinvention in which the projection device is separate from the lecturecapture hardware; and

FIG. 12 shows alternate connections to an overhead projector with amirror assembly consistent with the present invention.

DETAILED DESCRIPTION

Overview

Systems consistent with the present invention digitally capture lecturepresentation slides and speech and store the data in a memory. They alsoprepare this information for Internet publication and publish it on theInternet for distribution to end-users. These systems comprise threemain functions: (1) capturing the lecture and storing it into a computermemory or database, (2) generating a transcript from the lecture and thepresentation slides and automatically summarizing and outlining thetranscripts, and (3) publishing the lecture slides image data, audiodata, and transcripts on the Internet for use by client computers.

Generally, when the lecturer begins presenting, and the first slide isdisplayed on the projection screen by a projector, a mirror assemblychanges the angle of the light being projected on the screen for a briefperiod of time to divert it to a digital camera. At this point, thedigital camera captures the slide image, transfers the digital videoimage data to the computer, and the digital video image data is storedon the computer. The mirror assembly then quickly flips back into itsoriginal position to allow the light to be projected on the projectionscreen as the lecturer speaks. When this occurs, an internal timer onthe computer begins counting. This timer marks the times of the slidechanges during the lecture presentation. Simultaneously, the systembegins recording the sound of the presentation when the first slide ispresented. The digital images of the slides and the digital audiorecordings are stored on the computer along with the time stampinformation created by the timer on the computer to synchronize theslides and audio.

Upon each subsequent slide change, the mirror assembly quickly divertsthe projected light to the digital camera to capture the slide image ina digital form, and then it flips back into its original position toallow the slide to be displayed on the projection screen. The time ofthe slide changes, marked by the timer on the computer, is recorded in afile on the computer. At the end of the presentation, the audiorecording stops, and the computer memory stores digital images of eachslide during the presentation and a digital audio file of the lecturespeech. Additionally, it will have a file denoting the time of eachslide change.

Alternatively, in another implementation, slides can be generated usingmachines that are not conventional slide projectors. Acomputer-generated slide presentation can be used, thereby avoiding theneed of the mirror assembly and the digital camera. In the case of thecomputer-generated slide (e.g., PowerPoint™ available from MicrosoftCorporation of Redmond, Wash.), the digital video image data from thecomputer generating the slide is transferred to the system's computer atthe same time that the slide is projected onto the projection screen.Similarly, slides may be projected from a machine using overheadtransparencies or paper documents. This implementation also avoids theneed for the mirror assembly and the digital camera, because it, likethe computer generated presentations, transfer the video image datadirectly to the computer for storage at the same time that it projectsthe image onto the projection screen. Any of these methods or othermethods may be used to capture digital video image data of thepresentation slides in the computer. Once stored in the computer, thedigital video and audio files may be published to the Internet or,optionally, enhanced for more efficient searching on the Internet.

During the optional lecture enhancement, optical character recognitionsoftware is applied to each slide image to obtain a text transcript ofthe words on a slide image. Additionally, voice recognition software isapplied to the digital audio file to obtain a transcript of the lecturespeech. Once these transcripts are obtained, automatic summarization andoutlining software can be applied to the transcripts to create indexesand outlines easily searchable by a user. In addition to the enhancedfiles, the user will also be able to search the whole transcript of thelecture speech.

Finally, after transferring the files to a database, systems consistentwith the present invention publish these slide image files, audio filesand transcript files to the Internet for use by Internet clients. Thesefiles are presented so that an Internet user can efficiently search andview the lecture presentation.

Systems consistent with the present invention thus allow a lecturepresentation to be recorded and efficiently transferred to the Internetfor use by end-users. Whereas several implementations of the presentinvention are possible, some alternative embodiments are also discussedbelow.

System Description

FIGS. 1 and 2 illustrate hardware components in a system consistent withthe present invention. Although FIG. 1 shows an implementation with aslide projector, the system allows a presenter to use a variety of mediafor presentation: 35-mm slides, computer-generated presentations,overhead transparencies or paper documents. The overhead transparenciesand paper documents will be discussed below with reference to FIG. 4.

FIG. 1 demonstrates the use of the system with an integrated 35-mm slideprojector 100 that contains a computer. The output of the projectiondevice passes through an optical assembly that contains a mirror, asshown in FIG. 2. In the implementation shown in FIG. 1, the mirrorassembly 204 is contained in the integrated slide projector 100 behindthe lens 124 and is not shown on the FIG. 1. This mirror assembly 204diverts the light path to a chargecoupled device (CCD) 206 for a briefperiod of time so that the image may be captured. A CCD 206 is asolid-state device that converts varying light intensities into discretedigital signals, and most digital cameras (e.g., the Pixera ProfessionalDigital Camera available from Pixera Corporation of Los Gatos, Calif.)use a CCD for the digital image capturing process. The video signalcarrying the digital video image data from the CCD 206 enters a computer102, which is integrated within the projection box in thisimplementation, via a digital video image capture board contained in thecomputer (e.g., TARGA 2000 RTX PCI video board available from Truevisionof Santa Clara, Calif.). This system is equipped with a device (e.g.,Grand TeleView available from Grandtec UK Limited, Oxon, UK) thatconverts from SVGA or Macintosh computer output and allows forconversion of this signal into a format which can be captured by theTruevision card, whereas the Truevision card is accepts an NTSC(National Television Standards Committee) signal.

As the lecturer changes slides or transparencies, the computer 102automatically records the changes. Changes are detected either by aninfrared (IR) slide controller 118 and IR sensor 104, a wired slidecontroller (not shown) or an algorithm driven scheme implemented in thecomputer 102.

As shown in FIG. 2, when a slide change is detected either via the slidecontroller 118 or an automated algorithm, the mirror 208 of the mirrorassembly 204 is moved into the path of the projection beam at a45-degree angle. A solenoid 202, an electromagnetic device often used asa switch, controls the action of the mirror 208. This action directs allof the light away from the projection screen 114 and towards the CCD206. The image is brought into focus on the CCD 206, digitally encodedand transmitted to the computer 102 via the video-capture board 302(shown in FIG. 3 described below). At this point, the mirror 208 flipsback to the original position allowing the light for the new slide to bedirected towards the projection screen 114. This entire process takesless than one second, since the video capture is a rapid process.Furthermore, this rapid process is not easily detectable by the audiencesince there is already a pause on the order of a second betweenconventional slide changes. In addition, the exact time of the slidechanges, as marked by a timer in the computer, is recorded in a file onthe computer 102.

FIG. 3 depicts the computer 102 contained in the integrated slideprojector 100 in this implementation. It consists of a CPU 306 capableof running Java applications (such as the Intel Pentium (e.g., 400 MHzPentium II Processors) central processors and Intel Motherboards (Intel®N440BX server board) from Intel of Santa Clara, Calif.), an audiocapture card 304 (e.g., AWE64 SoundBlaster™ available from Creative Labsof Milpitas, Calif.), a video capture card 302, an Ethernet card 314 forinteraction with the Internet 126, a memory 316, a secondary storagedevice 310 (which may be a hard disk drive), and an infrared receiver312 to receive a slide change signal from the slide change controller118. The CPU 306 also has a timer 308 for marking slide change times,and the secondary storage device 310 contains a database 318 for storingand organizing the lecture data. Referring back to FIG. 1, the computer102 contains an integrated LCD display panel 106, and a slide-outkeyboard 108 used to switch among three modes of operation discussedbelow. For file storage and transfer to other computers, the computer102 also contains a floppy drive 112 and a high-capacity removable mediadrive 110, such as a Jaz™ drive available from Iomega of Roy, Utah. Thecomputer 102 may also be equipped with multiple CPUs 306, thus enablingthe performance of several tasks simultaneously, such as capturing alecture and serving a previous lecture over the Internet.

Simultaneously with the slide capturing, audio signals are recordedusing a microphone 116 connected by a cable 120 to the audio capturecard 304 which is an analog to digital converter in the computer 102,and the resulting audio files are placed into the computer's secondarystorage device 310.

In one implementation consistent with the present invention, thepresentation slides are computer-generated. In the case of acomputer-generated presentation, the video signal from the computer (notshown) generating the presentation slides is sent to a VGA to NTSCconversion device and then to the video capture board 302 before it isprojected onto the projection screen 114, thus eliminating the need todivert the beam or use the mirror assembly 204 or the CCD 206. This alsoresults in a higher-quality captured image.

FIG. 4 illustrates hardware for use in another implementation in whichoverhead transparencies or paper documents are used instead of slides orcomputer-generated images. Shown in FIG. 4 is an LCD projector 400 withan integrated digital camera 402, such as the Toshiba MediaStar TLP-511U. This projection device allows overhead transparencies and paperdocuments to be captured and converted to a computer video signal, suchas SVGA. This SVGA signal can then be directed to an SVGA-input cable404. In this case, the computer 102 detects the changing of slides viaan algorithm that senses abrupt changes in video signal intensity, andthe computer 102 records each slide change. As in the computer-generatedimplementation, the video signal is captured directly before beingprojected, (i.e., the mirror assembly 204 and CCD 206 combination shownin FIG. 2 is not necessary).

In one implementation, optical character recognition is performed on thecaptured slide data using a product such as EasyReader Elite™ fromMimetics of Cedex, France. Also, voice recognition is performed on thelecture audio using a product such as NaturallySpeaking™ available fromDragon Systems of Newton, Mass. These two steps generate text documentscontaining full transcripts of both the slide content and the audio ofthe actual lecture. In another implementation, these transcripts arepassed through outline-generating software, such as LinguistX™ fromInxight of Palo Alto, Calif., which summarizes the lecture transcripts,improves content searches and provides indexing. Other documents canthen be linked to the lecture (i.e., an abstract, author name, date,time, and location) based on the content determination.

These documents, along with the slide image information, are convertedto Web-ready formats. This audio, slide, and synchronization data isstored in the database 318 (e.g., Microsoft SQL) which is linked to eachof the media elements. The linking of the database 318 and other mediaelements can be accomplished with an object-linking model, such asMicrosoft's Component Object Model (COM). The information stored in thedatabase 318 is made available to Internet end-users through the use ofa product such as Microsoft Internet Information Server (IIS) software,and is fully searchable.

Methods and systems consistent with the present invention thus enablethe presenter to give a presentation and have the content of the lecturemade available on the Internet with little intervention. Whileperforming the audio and video capture, the computer 102 automaticallydetects slide changes (i.e., via the infrared slide device or anautomatic sensing algorithm), and the slide change information isencoded with the audio and video data. In addition, the Web-basedlecture contains data not available at the time of the presentation suchas transcripts of both the slides and the narration, and an outline ofthe entire presentation. The presentation is organized using both timecoding and the database 318, and can be searched and viewed using astandard Java™ enabled Web-interface, such as Netscape Navigator™. Javais a platform-independent, object-oriented language created by SunMicrosystems™. The Java programming language is further described in“The Java Language Specification” by James Gosling, Bill Joy, and GuySteele, Addison-Wesley, 1996, which is herein incorporated by reference.In one implementation, the computer 102 serves the lecture informationdirectly to the Internet if a network connection 122 is establishedusing the Ethernet card 314 or modem (not shown). Custom software,written in Java for example, integrates all of the needed functions forthe computer.

FIG. 5 shows, in detail, the ports contained on the back panel 500 ofthe integrated 35-mm slide projection unit 100 consistent with thepresent invention: SVGA-in 502, SVGA-out 502, VHS and SVHS in and out510-516, Ethernet 530, modem 526, wired slide control in 522 and out524, audio in 506 and out 508, keyboard 532 and mouse port 528. Inaddition, a power connection (not shown) is present.

Operation

Generally, three modes of operation will be discussed consistent withthe present invention. These modes include: (1) lecture-capture mode,(2) lecture enhancement mode, and (3) Web-publishing mode.

Capturing Lectures

FIG. 6 depicts steps used in a method consistent with the presentinvention for capturing a lecture. This lecture capture mode is used tocapture the basic lecture content in a format that is ready forpublishing on the Internet. The system creates data from the slides,audio and timer, and saves them in files referred to as “source files.”

At the beginning of the lecture, the presenter prepares the media ofchoice (step 600). If using 35-mm slides, the slide carousel is loadedinto the tray on the top of the projector 100. If using acomputer-generated presentation, the presenter connects the slidegenerating computer to the SVGA input port 502 shown in the I/O ports500. If using overhead transparencies or paper documents, the presenterconnects the output of a multi-media projector 400 (such as the ToshibaMediaStar described above and shown in FIG. 4) to the SVGA input port502. A microphone 116 is connected to the audio input port 506, and anEthernet networking cable 122 is attached between the computer 102 and anetwork outlet in the lecture room. For ease of the discussion tofollow, any of the above projected media will be referred to as“slides.”

At this point, the presenter places the system into “lecture-capture”mode (step 602). In one implementation, this is done through the use ofa keyboard 108 or switch (not shown). When this action occurs, thecomputer 102 creates a directory or folder on the secondary storagedevice 310 with a unique name to hold source files for this particularlecture. The initiation of the lecture capture mode also resets thetimer and slide counter to zero (step 603). In one implementation, threedirectories or folders are created to hold the slides, audio and timestamp information. Initiation of lecture capture mode also causes animmediate capture of the first slide using the mirror assembly 204 (step604). The mirror assembly 204 flips to divert the light path from theprojector to the CCD 206 of the digital camera. Upon the capturing ofthis first slide, the digital image is stored in an image format, suchas a JPEG format graphics file (a Web standard graphics format), in theslides directory on the secondary storage device 310 of the computer 102(i.e., slides/slide01.jpg). After the capturing of the image by the CCD206, the mirror assembly 204 flips back to allow the light path toproject onto the projection screen 114. The first slide is thenprojected to the projection screen 114, and the internal timer 308 onthe computer 102 begins counting (step 606).

Next, systems consistent with the present invention record the audio ofthe lecture through the microphone 116 and pass the audio signal to theaudio capture card 304 installed in the computer 102 (step 608). Theaudio capture card 304 converts the analog signal into a digital signalthat can be stored as a file on the computer 102. When the lecture iscompleted, this audio file is converted into the Active Streaming Formator RealAudio format for efficient Internet publishing. In oneimplementation, the audio signal is encoded into the Active StreamingFormat or RealAudio format in real time as it arrives and is placed in afile in a directory on the secondary storage device 310. Although, thisimplementation requires more costly hardware (i.e., an upgraded audiocard), it avoids the step of converting the original audio file into theInternet formats after the lecture is complete. Regardless, the originalaudio file (i.e., unencoded for streaming) is retained as a backup onthe secondary storage device 310.

When the presenter changes a slide (step 610) using the slide control118 or by changing the transparency or document, the computer 102increments the slide counter by one and records the exact time of thischange in an ASCII file (a computer platform and application independenttext format), referred to as the “time-stamp file”, written on thesecondary storage device 310 (step 512). This file has, for example, twocolumns, one denoting the slide number and the other denoting the slidechange time. In one implementation, it is stored in the time stampfolder.

Using the mirror assembly 204 (FIG. 2), the new slide is captured into aJPEG format graphics file (i.e., slide#.jpg, where # is the slidenumber) that is stored in the slides folder on the secondary storagedevice 310. When the new slide is captured, the mirror assembly 204quickly diverts the light from the slide image back to the projectionscreen 114 (step 616). If any additional slides are presented, theseslides are handled in the same manner (step 618), and the system recordsthe slide change time and captures the new slide in the JPEG graphicsfile format.

At the completion of the lecture, the presenter stops the “lecturecapture” mode with the keyboard 108. This action stops the timer andcompletes the lecture capturing process.

Enhancing Lecture Content

FIG. 7 depicts a flowchart illustrating a method for enhancing acaptured lectured consistent with the present invention. When thelecture is complete, and the system has all of the source filesdescribed above, in one implementation, it may enter “lectureenhancement mode.” In this mode, the system creates transcripts of thecontents of the slides and the lecture, and automatically categorizesand outlines these transcripts. Additionally, the slide image data filesmay be edited as well, for example, to remove unnecessary slides orenhance picture quality.

Initially, optical character recognition (OCR) is performed on thecontent of the slides (step 700). OCR converts the text on the digitalimages captured by the CCD 206 (digital camera) into fully searchableand editable text documents. The performance of the optical characterrecognition may be implemented by OCR software on the computer 102. Inone implementation, these text documents are stored as a standard ASCIIfile. Through the use of the time-stamp file, this file ischronologically associated with slide image data.

Similarly, voice recognition is performed on the audio file to create atranscript of the lecture speech, and the transcript is stored as anASCII file along with time-stamp information (step 702). The system alsoallows a system administrator the capability to edit the digital audiofiles so as to remove gaps or improve the quality of the audio usingproducts such as WaveConvertPro (Waves, Ltd., Knoxville, Tenn.).

Content categorization and outlining of the lecture transcripts isperformed by the computer 102 using a software package such asLinguistX™ from Inxight of Palo Alto, Calif. (step 704). The resultinginformation is stored as an ASCII file along with time-stampinformation.

Web Publishing

FIG. 8 is a flowchart illustrating a method for publishing a capturedlecture on the Internet consistent with the present invention. Afterlecture capture or enhancement, (step 800), the system may be set to“Web-publishing mode.” It should be noted that the enhancement of thelecture files is not a necessary process before the Web-publishing modebut simply an optimization. Also, note that for the Web-publishing modeto operate, a live Ethernet port that is Internet accessible must beconnected. Standard Internet protocols (i.e., TCP/IP) are used fornetworking. In this mode, all of the source files generated in thelecture capture mode, as well as the content produced in the enhancementmode, are placed in a database 318 (step 800). Two types of databasesmay be utilized: relational and object oriented. Each of these types ofdatabases is described in a separate section below.

Consistent with the present invention, the system obtains a temporary“IP” (Internet Protocol) address from the local server on the networknode to which the system is connected (step 802). The IP address may bedisplayed on the LCD panel display 106.

When a user accesses this IP address from a remote Web-browser, thesystem (the “server”) transmits a Java applet to the Web-browser (the“client”) via the HTTP protocol, the standard Internet method used fortransmitting Web pages and Java applets (step 804). The transmitted Javaapplet provides a platform-independent front-end interface on the clientside. The front-end interface is described below in detail. Generally,this interface allows the client to view all of the lecture content,including the slides, audio, transcripts and outlines. This informationis fully searchable and indexed by topic (such as a traditional table ofcontents), by word (such as a traditional index in the back of a book),and by time-stamp information (denoting when slide changes occurred).

The lecture data source files stored on the secondary storage device 310can be immediately served to the Internet as described above. Inaddition, in one implementation, the source files may optionally betransferred to external web servers. These source files can betransferred via the FTP (File Transfer Protocol), again using standardTCP/IP networking, to any other computer connected to the Internet. Theycan then be served as traditional HTTP web pages or served using theJava applet structure discussed above, thus allowing flexibility of useof the multimedia content.

Use of the Captured Lecture and the Front-End Interface

The end-user of a system consistent with the present invention cannavigate rapidly through the lecture information using a Java appletfront-end interface. This platform-independent interface can be accessedfrom traditional PC's with a Java-enabled Web-browser (such as NetscapeNavigator™ and Microsoft Internet Explorer™) as well as Java-enabledNetwork Computers (NCs).

FIG. 9 shows a front-end interface 900 consistent with the presentinvention. The front-end interface provides a robust andplatform-independent method of viewing the lecture content andperforming searches of the lecture information. In one implementation,the interface consists of a main window divided into four frames. Oneframe shows the current slide 902 and contains controls for the slides904, another frame shows the audio controls 908 with time information906, and a third frame shows the transcript of the lecture 910 andscrolls to follow the audio. The fourth frame contains a box in whichthe user can enter search terms 912, a pop-up menu with which the usercan select types of media they wish to search, and a button thatinitiates the search. Examples of search methodologies include:chronological, voice transcript, slide transcript, slide number, andkeyword. The results of the search are provided in the first threeframes showing the slides, the audio and the transcripts. In anotherimplementation consistent with the present invention, another window isproduced which shows other relevant information, such as relatedabstracts.

Description of the Database Structure

Before the source files generated in the lecture capturing process canbe published in a manner that facilitates intelligent searching, indexesto the source files must be stored in a database. The purpose of thedatabase is to maintain links between all source files and searchableinformation such as keywords, author names, keywords in transcripts, andother information related to the lectures.

There are two major methods for organizing a database that containsmultiple types of media (text, graphics and audio): object-oriented andrelational. An object-oriented database links together the differentmedia elements, and each object contains methods that allow thatparticular object to interact with a front-end interface. The advantageof this approach is that any type of media can be placed into thedatabase, as long as methods of how this media is to be indexed, sortedand searched are incorporated into the object description of the media.

The second method involving a relational database provides linksdirectly to the media files, instead of placing them into objects. Theselinks determine which media elements are related to each other (i.e.,they are responsible for synchronizing the related audio and slidedata).

FIG. 10 shows a schematic of a three-tier architecture 1000 used tostore and serve the multimedia content to the end-user. As shown in FIG.10, the database 318 comprises part of the three-tier architecture 1000.The database 318 (labeled as the “data tier”) is controlled by anintermediate layer instead of directly by the end-user's interface 1002(labeled as the “client tier”). The client is a computer running aWeb-browser connected to the Internet. The intermediate layer, labeledas the “application tier,” provides several advantages. One advantage isscalability, whereas more servers can be added without bringing down theapplication tier. Additionally, the advantage of queuing allows requestsfrom the client to be queued at the application tier so that they do notoverload the database 318. Finally, there is increased compatibility.Although the application tier and front-end are Java based, the database318 can communicate with the application tier in any manner whichmaximizes performance. The method of communication, protocols used, andtypes of databases utilized do not affect the communication between thebusiness logic and the front-end.

FIG. 10 also shows how the application tier consists of a MainProcessing Unit (MPU) 1004 and middleware 1020. On the MPU resides thecustom Java code that controls query processing 1008, managestransactions 1010 and optimizes data 1012. Additionally, this codeperforms OCR 1014 and voice recognition 1016 and encodes the media 1018.The middleware 1020 provides a link between the custom Java code and thedatabase 318. This middleware 1020 already exists as various mediaapplication programming interfaces (APIs) developed by Sun Microsystems,Microsoft, and others. The middleware 1020 abstracts the custom Javacode from the database 318.

The end-user or client interacts with the MPU 1004 within theapplication tier. In addition, information entering the database 318from the “lecture-capture mode” of the system enters at the applicationtier level as well. This information is then processed within the MPU1004, passed through the middleware 1020, and populates the database318.

Alternative Embodiments

There are many different methods of implementing a system that performsfunctions consistent with the present invention. Several alternativeembodiments are described below.

Separation into Different Units

In one embodiment consistent with the present invention, the system isseparated into several physical units, one for each mode (i.e., lecturecapture, enhancement and publishing). One physical unit is theprojection device and computer that contains all of the necessaryhardware to perform the lecture-capturing process. This hardwareincludes the mirror assembly, the CCD digital camera, a computer withvideo and audio capturing ability, an infrared sensing unit, andnetworking ability. In this implementation, the sole function of thisunit is to capture the lecture and create the source files on thesecondary storage of the unit. This capture device contains theprojection optics and can display either 35-mm slides, acomputer-generated presentation, overhead transparencies or paperdocuments.

In this implementation, the lecture enhancement activities are performedin a separate physical enclosure. This separate device contains acomputer with networking ability that performs the OCR, voicerecognition and auto-summarization of the source files generated in thelecture capturing process.

Finally, a third physical enclosure provides Web-publishing function andcontains a computer with network ability, a database structure andInternet serving software.

In this modular design, several categories of products can beenvisioned. One provides lecture capturing ability only and requiresonly the lecture-capturing devices. This system is responsible for thecreation and serving of the generated source files. Anotherimplementation provides lecture capturing and Web serving and onlyrequires the lecture-capturing devices and the Web-publishing devices.Yet another implementation adds the lecture-enhancement device to theabove set-up and also makes the lecture transcripts and summariesavailable to the Web.

Separation of the Mirror Assembly from the Projection Device andComputer

FIG. 11 depicts a lower-cost and even more modular way of providing thelecture-capturing functionality involving the separation of the mirrorassembly 204 and CCD 206 from the projection device. In this embodiment,the mirror assembly 204 and CCD 206 are a separate unit that snaps ontothe lens of the 35-mm slide projector 1102. As shown in FIG. 11, themirror assembly 204 and CCD 206 is connected by video cable 1104 to thecomputer 102, which sits in a separate box. This connection allows thecomputer 102 to receive digital video image data from the CCD 206 and tocontrol the action of the mirror 204 via the solenoid 202 (shown in FIG.2). The infrared beam from the slide controller 118 signals a slidechange to both the slide projector 1102 and the computer 102. Both theinfrared sensors on both devices are configured to receive the same IRsignal so that the slide controller 118 can control both devices. Forinstance, the slide projector 1102 may be purchased with a slidecontroller 118, in which case the slide projector 1102 will already betuned to the same infrared frequency as the slide controller 118, andthe infrared sensor in the computer 102 may be built or configured toreceive the same infrared frequency emitted by the slide controller 118.Such configuration of an infrared sensor to a particular frequency iswell known to those skilled in the art. Additionally, a computer monitor1110 is used in place of the LCD display. The advantage of this modularsetup is that once the appropriate software is installed, the user isable to use any computer and projection device desired, instead ofautomatically having them provided in the lecture-capturing boxdescribed above.

For capturing computer-generated presentations, the mirror assembly isnot used and the video signal and mouse actions from the user'sslide-generating computer pass through the capture computer before goingto the LCD projector. This enables the capture computer to record theslides and change times.

FIG. 12 shows another implementation using the connection of a separateCCD 206 and mirror assembly 204, described above, to a standard overheadprojector 1200 for the capture of overhead transparencies. A video cable1202 passes the information from the CCD 206 to the computer 27. Agooseneck stand 1204 holds the CCD 206 and mirror assembly 204 in frontof the overhead projector 1200.

Alternate Slide Capture Trigger

With the use of a Kodak Ektapro Slide Projector (Kodak, Rochester, N.Y.)which can either be incorporated into device 100 or used as astand-alone slide projector 1102, an alternative method of communicatingthe status of the slide projector to the computer 102 uses the P-Comprotocol (Kodak, Rochester, N.Y.). The P-Com protocol is communicatedbetween the slide projector and the computer 102 over an RS-232interface that is built into the Ektapro projector. The informationobtained from the projector provides the computer 102 with the datasignaling that a slide change has occurred and hence, the computer willthen digitally capture the slide. This alternative approach alleviatesthe need for detecting signals from the infrared controller 118 and IRsensor 104 or the wired slide controller.

Alternate Front-End Interfaces

Although the front-end interface described above is Java-based, if thevarious modes of operation are separated, alternate front-end interfacescan be employed. For example, if lecture capture is handled by aseparate device, its output is the source files. In this case, thesesource files can be transferred to a separate computer and served to theInternet as a web site comprised of standard HTML files.

In another implementation, the front-end interface can also be aconsumer-level box which contains a speaker, a small LCD screen, severalbuttons used to start and stop the lecture information, a processor usedto stream the information, and a network or telephone connection. Thisbox can approach the size and utility of a telephone answering machinebut provides lecture content instead of just an audio message. In thisimplementation, the lecture content is streamed to such a device througheither a standard telephone line (via a built-in modem for example) orthrough a network (such as a cable modem or ISDN). Nortel (Santa Clara,Calif.) provides a “Java phone” which can be used for this purpose.

Alternate Implementation of Application Tier

The system described in the Main Processing Unit (1004) and theApplication Programming Interface (1020) can be programmed using alanguage other than Java, e.g., C, C++ and/or Visual Basic Languages.

Alternate Optical Assembly for Image Capture

Another implementation of the present invention replaces the mirrorassembly 204 with a beam splitter (not shown). This beam splitter allowsfor slide capture at any time without interruption, but reduces theintensity of the light that reaches both the digital camera and theprojection screen 114.

If a beam splitter is used, redundancies can be implemented in theslide-capturing stage by capturing the displayed slide or transparency,for example, every 10 seconds regardless of the slide changeinformation. This helps overcome any errors in an automated slide changedetection algorithm and allows for transparencies that have been movedor otherwise adjusted to be recaptured. At the end of the lecture, thepresenter can select from several captures of the same slide ortransparencies and decide which one should be kept.

System Diagnosis

In one implementation consistent with the present invention, the usercan connect a keyboard and a mouse, along with an external monitor tothe SVGA-out port 504. This connection allows the user access to theinternal computer 102 for software upgrades, maintenance, and otherlow-level computer functions. Note that the output of the computer 102can be directed to either the LCD projection device or the LCD panel106.

Wireless Communications

In one implementation consistent with the present invention, the networkconnection between the computer and the Internet can be made usingwireless technology. For example, a 900 MHZ connection (similar to thatused by high quality cordless phones) can connect the computer 102 to astandard Ethernet wall outlet. Another option uses wireless cellularmodems (like those produced by Ricochet) for the Internet connection.

Electronic pointer

In another implementation, an electronic pointer is added to the system.Laser pointers are traditionally used by presenters to highlightportions of their presentation as they speak. The movement of thesepointers can be tracked and this information recorded and time-stamped.This allows the end-user to search a presentation based on the movementof the pointer and have the audio and video portion of the lecturesynchronized with the pointer.

Spatial positional pointers can also be used in the lecture captureprocess. These trackers allow the system to record the presenter'spointer movements in either 2-dimensional or 3-dimensional space.Devices such as the Ascension Technology Corporation pcBIRD™ or 6DOFMouse™ (Burlington, Vt.), INSIDETRAK HP by Polhemus Incorporated(Colchester, Vt.), or the Intersense IS-300 Tracker from Intersense(Cambridge, Mass.) can be used to provide the necessary trackingcapability for the system. These devices send coordinate (x, y, z) datathrough an RS-232 or PCI interface which communicates with the CPU 306,and this data is time-stamped by the timer 308.

Conclusion

Methods and systems consistent with the present invention provide astreamlined and automated process for digitally capturing lectures,converting these lectures into Web-ready formats, providing searchabletranscripts of the lecture material, and publishing this information onthe Internet. The system integrates many different functions into anorganized package with the advantages of lowering overall costs ofInternet publishing, speeding the publishing process considerably, andproviding a fully searchable transcript of the entire lecture. Since thelecture is ready for publishing on the Web, it is viewable on anycomputer in the world that is connected to the Internet and can use aWeb browser. Additionally, anyone with an Internet connection may searchthe lecture by keyword or content.

The foregoing description of an implementation of the invention has beenpresented for purposes of illustration and description. It is notexhaustive and does not limit the invention to the precise formdisclosed. Modifications and variations are possible in light of theabove teachings or may be acquired from practicing of the invention. Thescope of the invention is defined by the claims and their equivalents.

What is claimed is:
 1. A method of capturing a live presentation,comprising the steps of: capturing still images from a display devicewhich displays said still images for viewing by an audience during alive presentation; during the live presentation, detecting the changeover form one still image to another; recording the audio portion of aspeaker's presentation during a live presentation; and in response tosaid detected change over from one still image to another, automaticallysynchronizing change over from one still image to another with the audiorecording.
 2. A method of capturing a live presentation, comprising thesteps of: capturing still images from a display device which displayssaid still images for viewing by an audience during a live presentation,wherein the step of capturing still images from the display deviceincludes the steps of diverting light image from the display device to adigital imaging device; during the live presentation, detecting thechange over form one still image to another; recording the audio portionof a speaker's presentation during a live presentation; and in responseto said detected change over from one still image to another,automatically synchronizing change over from one still image to anotherwith the audio recording.
 3. A method of capturing a live presentation,comprising the steps of: capturing still images from a display devicewhich displays said still images for viewing by an audience during alive presentation, wherein the step of capturing still images includesdiverting a portion of the light still image from the display deviceonto a digital imaging device; during the live presentation, detectingthe change over form one still image to another; recording the audioportion of a speaker's presentation during a live presentation; and inresponse to said detected change over from one still image to another,automatically synchronizing change over from one still image to anotherwith the audio recording.
 4. A method of capturing a live presentation,comprising the steps of: capturing still images from a display devicewhich displays said still images for viewing by an audience during alive presentation, wherein the step of capturing still images includesintegrating an image diverting and capture device in front of a stillimage projection device; during the live presentation, detecting thechange over form one still image to another; recording the audio portionof a speaker's presentation during a live presentation; and in responseto said detected change over from one still image to another,automatically synchronizing change over from one still image to anotherwith the audio recording.
 5. A method in according with claim 1, furthercomprising the step of displaying a still image on the display devicewhich displays said still images for viewing by an audience.
 6. A methodin accordance with claim 5, wherein the displaying step includes thesteps of presenting a slide on an image projection device, said slideoriginating from one of a computer program, a negative film, an overheadtransparency and an opaque paper projection device.
 7. A method inaccordance with claim 1, wherein the step of recording includes the stepof picking up an audio signal from a microphone adjacent to the persongiving the live presentation.
 8. A method in accordance with claim 1,wherein the step of automatically synchronizing change over one stillimage to another still image with the audio recording includes manualinput of the change over event.
 9. A method in accordance with claim 1,wherein the step of automatically synchronizing change over one stillimage to another still image with the audio recording includesautomatically detecting the change over event.
 10. A method inaccordance with claim 1, further comprising the steps of determining thelocation of an input device pointer on the display device; andassociating a time stamp with a determined location, wherein theautomatic synchronizing step further includes the step of storing thedetermined location of the pointer and the associated time stamp intomemory.
 11. A method in accordance with claim 1, further comprising thesteps of storing the captured still images in a database; and providingsearch capabilities for searching the database.
 12. A method inaccordance with claim 10, further comprising the step of creating asearchable transcript of text in the still images.
 13. A method inaccordance with claim 11, wherein the step of creating a transcriptincludes optical character recognition.
 14. A method in accordance withclaim 11, further comprising the step of auto-summarizing the transcriptto generate an a summary of the transcript.
 15. A method in accordancewith claim 11, further comprising a step of auto-outlining thetranscript to generate an outline of the transcript.
 16. A method inaccordance with claim 1, further including the step of transmitting saidcaptured still images and recorded audio portion of a presentation to anetwork in a format suitable for viewing over the network.
 17. A methodin accordance with claim 16, further including the step of sending thecaptured still images and audio recording to a client via the Internet.18. A method in accordance with claim 17, further including the step ofconverting the audio recording of the live presentation into a streamingformat for transfer via the Internet.